GENERATING STRUCTURED DRUM PATTERN
USING VARIATIONAL AUTOENCODER AND SELF-SIMILARITY MATRIX

Supplementary Audio Files and Code

I-CHIEH Wei $^{1}$ , Chih-Wei Wu $^{2}$ , Li Su $^{1}$

^{1}

$^{1}$ Institute of Information Science, Academia Sinica, Taiwan

^{2}

$^{2}$ Netflix, Inc., USA

^{3}

$^{3}$ Netflix, Inc., USA

y i n j y u n_l u o @ m y m a i l . s u t d . e d u . s g, k a t_a g r e s @ i h p c . a - s t a r . e d u . s g, d o r i e n_h e r r e m a n s @ s u t d . e d u . s g

$\tt helloworld1033@iis.sinica.edu.tw, chihweiw@netflix.com, iis.sinica.edu.tw@iis.sinica.edu.tw$

GENERATING STRUCTURED DRUM PATTERN USING VARIATIONAL AUTOENCODER AND SELF-SIMILARITY MATRIX

Drum pattern generation is a task that focuses on the rhyth mic aspect of music and aims at generating percussive sequences.

audio synthesis.

Firstly, we present the audio in this work.

French horn
The original audio waveform

The resynthesized audio waveform from the Mel-spectrogram using Grin-Lim

Piano
The original audio waveform

The resynthesized audio waveform from the Mel-spectrogram using Grin-Lim

Cello
The original audio waveform

The resynthesized audio waveform from the Mel-spectrogram using Grin-Lim

Basson
The original audio waveform

The resynthesized audio waveform from the Mel-spectrogram using Grin-Lim

Now we demonstrate the controllable sound synthesis. As described in the paper Section 4.3, we specify the target pitch $\mathbf{y_m}$ and instrument $\mathbf{y_k}$ , and sample the pitch code $\mathbf{z}_p$ and timbre code $\mathbf{z}_t$ from the conditional distribution $p(\mathbf{z}_p | \mathbf{y_m})$ and $p(\mathbf{z}_t | \mathbf{y_k})$ , respectively, where $p(\mathbf{z}_{p} | \mathbf{y}_{p}) = \mathcal{N}(\mathbf{\mu}_{\mathbf{y}_{p}}, \textrm{diag}(\mathbf{\sigma}_{\mathbf{y}_{p}}))$ and $p(\mathbf{z}_{t} | \mathbf{y}_{t}) = \mathcal{N}(\mathbf{\mu}_{\mathbf{y}_{t}}, \textrm{diag}(\mathbf{\sigma}_{\mathbf{y}_{t}}))$ . In the following demonstration, we specify the same pitches for all instruments, play the audio and display the corresponding Mel-spectrograms.

English horn

French horn

Tenor Trombone

Trompet

Piano

Violin

Cello

Saxophone

Bassoon

Clarinet

Many-to-Many timbre transfer

In this section, we demonstrate the model's applicability in timbre transfer.

As described in Section 4.4 in the paper, we first infer $\mathbf{z}_p$ and $\mathbf{z}_t$ of the source input, and modify $\mathbf{z}_t$ (denoted as $\mathbf{z}_{source}$ ) by:

z_{t r a n s f e r} = z_{s o u r c e} + α μ_{s o u r c e \to t a r g e t},

$\mathbf{z}_{transfer} = \mathbf{z}_{source} + \alpha\mathbf{\mu}_{source \rightarrow target},$
where

μ_{s o u r c e \to t a r g e t} = μ_{t a r g e t} - μ_{s o u r c e}

$\mathbf{\mu}_{source \rightarrow target} = \mathbf{\mu}_{target} - \mathbf{\mu}_{source}$ , and

α \in [0, 1]

$\alpha \in [0, 1]$ . We then synthesize the spectrogram by passing

[z_{p}, z_{t r a n s f e r}]

$[\mathbf{z}_p, \mathbf{z}_{transfer}]$ to the decoder. See Fig. 4 for an illsutration of transferring French horn to piano.

Note that, in practice, we do not need labels of source instrument and pitch for timbre transfer, as the two variables are automatically inferred by $q(\mathbf{z}_p | \mathbf{X})$ and $q(\mathbf{z}_t | \mathbf{X})$ , respectively. $q(\mathbf{y}_{t} | \mathbf{X})$ infers the mixture component (source instrument identity) to which $\mathbf{X}$ belongs, and $\mathbf{\mu}_{source \rightarrow target}$ is then obtained by subtracting mean of the mixture component of the target to the that of the source.

Following Fig. 5 in the paper, we demonstrate $\texttt{Fhn} \rightarrow \texttt{Pno}$ , $\texttt{Pno} \rightarrow \texttt{Vc}$ , $\texttt{Vc} \rightarrow \texttt{Bn}$ , and $\texttt{Bn} \rightarrow \texttt{Fhn}$ .
The source instrument is gradually changed to the target instrument, by $\alpha=\{0, 0.25, 0.5, 0.75, 1.0\}$ .